Council Member Vacca (photo: @JamesVacca13)

There are two years to go before the deadline set by the city’s 2012 Open Data Law, which requires city agencies identify and publish all their citywide aggregated public data online. At a Wednesday afternoon oversight hearing of the City Council’s Committee on Technology, open data and civic technology advocates offered praise for the city’s progress thus far and recommendations for the city to meet its goals.

Open data is considered a vital transparency tool that allows citizens and government to evaluate and improve city services. New York City, considered a trendsetter in this area, committed to this goal in 2012 by passing a law that would make data easily accessible and consumable by the public. Through the city’s Open Data Portal, New Yorkers can access a range of information such as the number and type of complaints made to the 311 helpline, real-time traffic speed data, or where the City Council spends discretionary funding.

The Open Data Portal hosts nearly 1,600 datasets across all city agencies.

Last year, in the annual progress report on the city’s implementation of the Open Data Law,  Mayor Bill de Blasio released a new open data plan, “Open Data For All,” which aims at making the Open Data Portal more user-friendly, making it easily visualized for those with little experience in data or programming, and pushes for more citywide engagement.

Following that, the City Council passed a package of legislation to enhance the existing law and improve access to information through the city’s Open Data Portal maintained by the Department of Information Technology and Telecommunications (DoITT).

This was the first year which saw some of the new amendments put into effect. These new laws, seven in total, deal with retention of data, building data dictionaries, standardizing geospatial data, establishing timelines for agencies to respond to public requests on the Open Data Portal, mandating timely updates for public data, requiring publication of datasets obtained through Freedom of Information Law requests, and examination and verification of agency compliance.

“One of the things we’ve been able to do is build an open data ecosystem,” testified Dr. Amen Ra Mashariki, director of the Mayor’s Office of Data Analytics (MODA), who is also the chief analytics officer and chief open platform officer for the city. He said the new laws help “anchor the administration’s commitment to transparency and equitable uses of technology around open data.”

Ra Mashariki stressed that MODA’s focus has been on user research, feedback mechanisms, and technical standards, and he laid out the city’s progress on each while teasing updates that will be released later this year. These updates include a first-of-its-kind analysis with NYU Center for Urban Science and Progress on “data poverty” -- the lack of access or representation of communities in city data; a study with Columbia University’s School of International and Public Affairs on opportunities for community based organizations to utilize open data; and a new centralized mechanism for receiving feedback, with quicker responses and tracking.

Concluding his testimony, Ra Mashariki cited an important example of how open data can be useful to the public. He referenced Ben Wellington, a data scientist who runs the blog “I Quant NY.” Wellington examined parking violation data and found that the NYPD had been ticketing legally parked cars for millions of dollars. He took his analysis to the NYPD who confirmed it and took measures to prevent it in the future.

Testifying on behalf of DoITT, Albert Webber cited the city’s key accomplishments over the last year -- traffic on the Open Data Portal increased to more than 5 million hits; 116 datasets were made public, bringing the total to nearly 1,600; 100 new data sets are now automatically updated; and more than 40 unscheduled data sets were identified and published. He also said DoITT would soon fill three of four new dedicated positions for the Open Data Portal.

At the hearing, government transparency advocates and civic technology professionals wholeheartedly endorsed the city’s efforts, and also made a few recommendations for improvements.

John Kaehny, executive director of Reinvent Albany and co-chair of the New York City Transparency Working Group, emphasised “that City Council stay actively and energetically involved in pushing, cajoling and cheerleading for open data in this city,” without which the program can not be a success.

The Transparency Working Group is co-chaired by Gene Russianoff, senior attorney at the New York Public Interest Research Group, and its members include Common Cause New York and Citizens Union. “Open data and the idea behind open data is working in New York City,” Kaehny said, also stressing that city agencies will eventually save tens of millions of dollars by using open data.

He said the de Blasio administration has “started upping their game” in implementing open data but in his testimony Kaehny pointed out a number of outstanding issues. These included the need for a better process for remedying public complaints about data quality, more standardization of data, clearer agency criteria for publishing data, and publishing high-demand data sets, such as NYPD crime statistics or the Department of Transportation’s street paving schedule.

He also suggested other needed improvements to the overall system and the law. Currently, one of the new laws requires publishing the number of datasets requested under FOIL -- a method often used by advocates and journalists to examine city information. These, he said, should also be named. He said the city should also consider creating a data issues tracker to monitor errors in datasets reported by the public, similar to a tool used by the federal government.

Council Member James Vacca, chair of the technology committee, was generally satisfied with the administration although he raised a few concerns at the hearing. He was especially pleased that one dataset to be added soon will be of snowplow locations, updated every hour, which would give people quick answers during snowstorms when the city usually struggles to clear roads across the five boroughs. He did question the administration on whether certain agencies were fully complying with the law, particularly the New York Police Department.

Vacca also wondered later whether open data use was being taught to local community groups and Community Boards, which would then have a stronger ability to advocate for local issues. “I know communities that always say ‘We want more police, we want more traffic agents...but they don’t have the documentation, they don’t do the research,” he said. “Here it’s at their fingertips, here we can do the research.”

DoITT’s Webber said that agencies “have made strides across the board,” and that the NYPD had added a number of datasets to its website and the Open Data Portal links to them. “Overall I wouldn’t say any particular agency is not committed to doing what they’re supposed to do,” Webber said, adding that he would later follow up with the Council Member with specific NYPD datasets.

For civic technology advocates, the major concern is not only the availability of data but its usefulness to the public.

Noel Hidalgo, executive director of BetaNYC, a civic technology and open government group, made this clear even as he praised the administration and the Council for their push on open data. “The seven pieces of legislation that were passed are taking us in the right direction and have really strengthened the city’s open data practice,” he said. But Hidalgo pointed out that one big weakness, echoing Vacca’s words, is figuring out how to teach community groups to access, understand and utilize open data. “Frustratingly, there are zero best practices out there on how to teach the city’s open data to itself,” Hidalgo said.

Hidalgo added that under the provisions of the new laws passed last year, the data dictionaries need to contain tutorials and that there needs to be easier access to bulk data downloading. Finally, he stressed the need for the city to create an open data legacy that can last through successive administrations and City Council classes, with a unit dedicated to leadership, standardization, development, and education.

There was also consensus at the hearing among advocates that the very infrastructure of the Open Data Portal could be modernized. Joel Natividad, director of open data at OpenGov Inc., said the portal, currently run by a contracted vendor, needs to be an open source platform that can “support innovation and experimentation” by universities, non-governmental organizations and the private sector. This will allow the city to “tap the genius of the crowd,” he said. Kaehn, of Reinvent Albany, agreed, saying the current platform has severe limitations.